Skip to content

feat(verify-pyprocar): make fetched fixtures read-only and keep verify.sh inside data/ - #303

Merged
lllangWV merged 41 commits into
devfrom
read-only-fixtures
Oct 5, 2026
Merged

lllangWV merged 41 commits into
devfrom
read-only-fixtures

Conversation

@lllangWV

@lllangWV lllangWV commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Why

Fixtures in data/ are writable, so only the test audit hook stops a stray write, and agent briefs repeat the ban by hand (#288). This PR makes verify.sh fetch lock each fixture read-only, and it keeps every write, chmod and delete that verify.sh runs inside the main checkout's data/.

Round 1 of this PR failed review on that second promise. A verifier pointed a scratch data symlink at / and ran fetch data/home and fetch data/tmp. Two bugs let the round-1 code run chmod a-w on about 2 million of the operator's files, since restored:

  • A. real_path ignored a failed cd -P. When data exists but cannot be entered, real_path data printed an empty string, so "$data"/?* became /?* and fixture_path data/home returned /home.
  • B. When data resolves to /, bash prints cd -P data/tmp && pwd as //tmp, and "$data"/?* became //?*, which matches.

Round 2 failed on a write rather than a chmod. run named its dir $RUNS/<timestamp>-<name> and wrote into it with mkdir -p, echo $$ >.pid and cp -RL without checking that the dir was new. An entry planted at that name made run write outside data/ before chmod_below_data refused (exit 2). The entry could be a symlink, or a real run dir whose work, .pid or evidence links out. The EXIT trap then deleted the outside .pid. origin/dev has the same defect, and there run exits 0.

What changed

The data root is now one fixed place, checked before any command touches data/:

  • require_data (verify.sh) resolves data and accepts it only when it equals <main checkout>/data, with data/verify-runs a real directory in it. In the main checkout that means the directory itself; in a linked worktree, the symlink worktree-setup makes. doctor, fetch, run, clean and gc exit 2 otherwise. An empty root, /, //, an unenterable directory or a copy elsewhere can never equal it.
  • real_path returns non-zero on any failed cd, and physical_dir strips a leading //.
  • is_below compares canonical paths component by component, and needs the child strictly below the parent. fixture_path uses it against the data root, clean against verify-runs.
  • Every chmod goes through chmod_below_data, and every rm -rf through scrub. Both first call require_resolved_below, which refuses a path that is not already resolved and strictly below the data root (scrub: below verify-runs). The chmod never follows a symlink and skips files with a second hard link (D). fetch names each skipped file on stderr, and doctor keeps counting it as writable, so the operator sees it.
  • download_from_hf fetches exactly <relpath>.zip with snapshot_download(local_dir=<staging dir beside the target>) and raises FileNotFoundError when the dataset has no such archive, so fetch data/c downloads nothing (C). This also removes two deletes outside data/ that the helper ran: rmtree of the dataset's folder in the active Hugging Face cache, and rmtree <output_path>/.cache, which for fetch was the repo root's .cache.
  • clean, run and gc use the resolved verify-runs, so a verify-runs symlink out of data/ is refused (E).
  • doctor lists every root that fetch refuses to lock (a dot name, a space, a symlink out of data/) under unlockable fixture roots (F).
  • run creates its run dir with a plain mkdir "$run", which fails on any existing entry, including a dangling symlink, and then exits 2. The dir must resolve strictly below verify-runs (require_resolved_below). Its subdirs use plain mkdir, .pid is written with noclobber, and the trap that removes .pid is set only after this process made the dir. Every later write goes into that new dir. This also stops two runs with one name in the same second from sharing a dir: the second is refused and can be rerun (round 3, finding A).
  • doctor exits 2 like the other data commands when the data root is wrong. Before, it printed the refusal and exited 0.
  • The chmod lint is replaced by a sweep test that runs every command against a sacrificial tree and compares everything outside data/ before and after. A census lint lists every line of verify.sh that writes, links, moves, deletes or changes modes, and fails when one is added or changed. It matches chmod, chown, chgrp, chattr, setfacl, install (any form), rm, rmdir, unlink, mv, shred, cp, ln, mkdir, touch, tee, truncate, dd, rsync, mktemp, mkfifo, mknod, tar, unzip, sed -i, perl -i, find -delete, the download call, a command held in a variable ($w f, "$@", "${cmd[@]}", if "$w"), and every redirect to a file except /dev/null: >, >>, >|, <>, N>, &>, >&file and exec N> (G, widened in round 4). test_the_census_flags_each_way_a_line_can_write appends each form to verify.sh and checks that the census grows by that line, and that 2>/dev/null, >&2, 2>&1, exec 3>&- and a > inside quotes do not count. Python writes inside py -c code and git writes are outside the lint; the sweep test is the behavioural backstop.
  • The CLAUDE.md row now says what verify.sh writes: only below the data root, plus the writes SKILL.md lists outside it (round 4, finding B). doctor, a download in fetch and a run driver import pyprocar, which appends to <repo>/pyprocar.log. A download writes the dataset's refs/main in the shared cache even with local_dir (_snapshot_download.py:369-376 at 1.2.3), and hf_xet writes a log to $HF_HOME/xet/logs/ and creates chunk-cache/ and staging/ under $HF_HOME/xet/<endpoint>/. In the main checkout pixi run --locked can install or update the env but never rewrites pixi.lock. worktree-setup and exec write their own worktree's _version.py, .tmp and data link, and worktree-setup's git diff can refresh that worktree's index. SKILL.md says the same, names the leading . and - that plain names exclude, and shows how the operator unlocks a fixture to refresh it. SKILL.md replaces the raw chmod -R u+w before worktree remove with find -P .tmp -type d -exec chmod u+w {} + on named copies, never data (H).
  • Round 5: tests copy a fixture only with tests.utils.writable_copy, which now also copies a single file, and tests/test_fixture_copies.py enforces it (see Round 5). verify.sh exports PYTHONDONTWRITEBYTECODE=1. SKILL.md and CLAUDE.md list matplotlib's config dir and font cache among the writes outside data/.
  • Round 4: fetch runs the download with HF_XET_LOG_DIR_DISABLE_CLEANUP=1, because hf_xet's log cleanup otherwise deletes older logs in $HF_HOME/xet/logs/. In the main checkout py runs pixi run -q --locked -e default, as exec already did, so it never re-locks. doctor runs git status with GIT_OPTIONAL_LOCKS=0, so it no longer rewrites the git index (B).

Census

Every line of verify.sh that writes, overwrites, links, moves, deletes or changes modes. test_verify_sh_writes_only_on_the_lines_its_census_reviewed lists the same lines. "Resolved" means real_path gives the path back unchanged: no symlink in it, no . or ...

Line Command Target Why it stays inside
PATH=... "$@" shared_env (exec, and py in a worktree) whatever the command writes runs the caller's own command or python; exec is the CI-gate runner and writes what that tool writes
[ -x "$SHARED_ENV/python" ] || { echo "...run 'pixi install -e dev'..." } doctor, run, exec, worktree-setup none flagged only because the message names pixi install; it prints and exits 2
find -P "$2" ... -exec chmod "$1" {} + chmod_below_data (fetch, run) $2 and below require_resolved_below "$2" "$DATA" first; find -P never follows a link; skips links and files with a second hard link
rm -rf "$1/work" "$1/.start" "$1/.pid" scrub (clean, gc) entries of a run dir require_resolved_below "$1" "$RUNS" first; rm -rf removes a symlink entry without following it
pyprocar.download_from_hf(...) fetch data/examples/<cat>/<name> only when the fixture is absent and fixture_path puts it strictly below the resolved data root; stages in a new temp dir beside it; the documented exceptions are the refs/main file, hf_xet's log and caches (cleanup off) and pyprocar.log
mkdir -p "$RUNS" run <data>/verify-runs require_data checked that it resolves to exactly $DATA/verify-runs
mkdir "$run" run $RUNS/<stamp>-<name> new: a plain mkdir fails on any existing entry; <name> is one plain name
mkdir "$run/evidence" "$run/work" "$run/work/tmp" run inside the new run dir new entries in a dir this process just made, after require_resolved_below "$run" "$RUNS"
(set -C; echo $$ >"$run/.pid") run $run/.pid new; noclobber refuses an existing file
trap 'rm -f "$run/.pid"' EXIT run $run/.pid set only after this process made $run; rm -f removes a link without following it
cp -RL ... "$fixture" "$run/work/calc" run $run/work/calc calc does not exist yet, so every destination file is new; the source is a resolved fixture
cp "$driver" "$run/evidence/driver.py" run new file in the new run dir
touch "$run/.start" run new file in the new run dir
py ... >"$run/evidence/run.log" run new file opened before the driver starts
echo "$code" >"$run/evidence/exit_code" run evidence after the driver, which is the user's own code and can write anywhere itself; nothing in verify.sh creates this file earlier
(cd calc && find ...) >"$run/evidence/side_effects.txt" run evidence as above
{ echo "missing $MAIN/pyprocar/_version.py; run 'pixi install -e dev'..." } worktree-setup none flagged only because the message names pixi install; it prints and exits 2
cp "$MAIN/pyprocar/_version.py" pyprocar/_version.py worktree-setup this worktree's file worktree-own exception; it follows a symlinked _version.py (pre-existing, see Scope)
mkdir -p "$MAIN/data" .tmp worktree-setup main data/, this worktree's .tmp worktree-own exception; mkdir -p creates nothing when the path exists
ln -sfn "$MAIN/data" data worktree-setup this worktree's data link refused earlier when data is a real directory; -n replaces the link itself
mkdir -p .tmp exec this worktree's .tmp worktree-own exception

The library side of fetch (download_test_data): mkdir(parents=True, exist_ok=True) of the category dir, TemporaryDirectory(dir=<category dir>), replace of the archive into the category dir, extraction into the new fixture dir, and unlink of the archive. All of these are inside the target's category dir. The library no longer runs rmtree.

Scope

  • Covers verify-pyprocar: make fetched fixtures read-only at the OS level #288 and round-1 findings A to I. The shared data/ stays writable until the operator locks it once after merge. Run the lock from a checkout at merged dev (a linked worktree after verify.sh worktree-setup, or the main checkout once it is at merged dev), because the main checkout today is at an older commit whose fetch locks nothing. Run it from that checkout's root, because the shell expands data/examples/*/* from the working directory: .claude/skills/verify-pyprocar/scripts/verify.sh fetch data/examples/*/* data/codes data/io data/issues. verify.sh doctor then prints writable fixture paths: 0. To refresh one fixture later, unlock its directories from the main checkout's root with find -P data/examples/<category>/<name> -type d -exec chmod u+w {} +, remove it, and fetch it again. rm -rf needs write permission on directories only, so this never unlocks a file whose second hard link may be outside data/ (round 4, finding D). SKILL.md shows the same three commands. download_test_data(force=True) on a locked fixture raises PermissionError until it is unlocked.
  • The main checkout's data/ must be a real directory now. A data symlink in the main checkout (say, to another disk) is refused, because it cannot equal <main>/data. SKILL.md says so.
  • tests/scripts/download_test_data.py calls download_test_data("data") and scripts/download.py shows a category prefix in its epilog. Both now raise FileNotFoundError. Both already crashed on dev: the first on the missing data.zip after copying every archive, the second on str / str because it passes output_path as a string. They are left for a follow-up.
  • cp --reflink=auto in run is GNU-only, as before. CI runs on Ubuntu only.
  • Pre-existing and left as is (reported for follow-up): worktree-setup's cp writes through a symlinked worktree pyprocar/_version.py; a newline in the main checkout's path breaks run's du line (exit 1); doctor truncates an unlockable name that holds a newline; a library caller who passes glob characters to download_from_hf downloads every matching archive into staging and then raises.

Blast Radius

Skill scripts, download_from_hf, two test helpers, the test modules that copy fixtures, and a new static check on tests/. download_from_hf now raises FileNotFoundError for a relpath that is not exactly one archive in the dataset; before, a category path such as data/examples/bands downloaded every archive in it and then crashed on the missing bands.zip. After the operator locks data/, code that writes into a fixture fails with PermissionError instead of changing shared data.

Verification

Round 5 (head bf6b7a1 on dev 499b1bc)

Safety as in rounds 3 and 4. Every run of verify.sh, its tests and its mutants used bwrap --ro-bind / / --bind <fake root> --tmpfs /tmp --unshare-net, with the fake root under the worktree's .tmp. Gates and the lint ran in the overlay sandbox, where worktree writes go to an invisible tmpfs. The shared data/ was only read. It still has 2230 writable entries under codes, examples, io and issues.

Finding (blocking): #304's Elk test on locked fixtures. The branch is rebased onto dev 499b1bc (#304 and #305). In test_real_elk_6_3_bands_beside_a_task_10_elmirep_keep_their_l_m_names (tests/pyprocar/io/elk/test_parser.py), copytree kept the 0555 dir mode, so the next shutil.copy of ELMIREP.OUT raised PermissionError. Both copies now go through writable_copy, which also copies a single file now (shutil.copyfile, so the new file gets the default mode). test_auto_bandsplot.py's auto_calc fixture also uses writable_copy. It passed when locked, because it writes nothing into the copy, but the new rule allows no raw fixture copy.

The lesson as a check: tests/test_fixture_copies.py. This is the second merged PR that added a raw fixture copy, and CI has no data/, so the check reads the source. It fails on shutil.copy, copy2, copyfile, copytree, copymode, copystat, os.link, Path.hardlink_to, or cp or rsync through subprocess or os.system, anywhere in tests/ outside writable_copy, when an argument holds a fixture path. A fixture path is built from DATA_DIR, or from __file__ or ROOT_DIR joined with "data". The check follows it through names (by scope), attributes, defaults, call arguments, return values, pytest fixtures (conftest included), parametrize values and imports across modules. The result of writable_copy is not a fixture path. Each message ends with copy it with tests.utils.writable_copy. The CLAUDE.md rule table has the rule and the enforcer.

  • On the head's tests/: 0 hits. At the first lint commit (2beef1c, before the fix): 3 hits, elk/test_parser.py:862 (copytree), :867 (copy) and test_auto_bandsplot.py:52.
  • From git history, with the head's tests/utils: f93c8bdf:tests/pyprocar/io/elk/test_parser.py gives 2 hits (lines 862 and 867), 60fc0c02:tests/pyprocar/scripts/test_filtered_procar.py gives 1 hit (line 43, shutil.copy(src / name, ...) in _copy_calc, reached through the src=CALC default). Both files at the head give 0.
  • The self-test runs the check on synthetic modules. It covers 22 flagged forms and the forms that must pass: a command that is not a copy, a copy of other paths, rmtree, a copy out of a writable_copy result, a "data" dir under a temp path, a shadowing parameter, and writable_copy's own body. It also holds verbatim excerpts of both historical sites. Fourteen mutants of the check are all killed: no helper exemption, writable_copy not a sanitizer, no repo-data seed, and no defaults, call arguments, returns, cross-module exports, fixtures, attributes, shell, hardlink_to, parametrize or loops, and scope lookup that ignores shadowing.
  • A bare "data" path component is not a seed, because test_verify_fixtures.py builds sacrificial data/ trees under tmp_path. A seed of that kind flagged three of its lines.

Locked run of the two sites (private copy locked by this PR's fetch, -k on the Elk and auto_calc tests): at 2beef1c, 1 failed (PermissionError: ... bands/ELMIREP.OUT), 2 passed. At 5222718, 3 passed.

Full -m data -n 4 on locked fixtures, at bf6b7a1. This PR's fetch locked a private reflink copy of data/{codes,examples,io,issues} in a fake main checkout. Only the 9 container dirs stayed writable. A linked worktree at the head, whose data links to that copy, ran the suite in the overlay sandbox: 411 passed, 1 skipped, 0 failed.

Round-4 notes, fixed.

  • __pycache__: measured in a sandbox with an overlay on the main checkout. After a touch of pyprocar/core/ebs.py, pixi run -q --locked -e default python -c 'import pyprocar.core.ebs' rewrote ebs.cpython-312.pyc. With PYTHONDONTWRITEBYTECODE=1 it left the file alone. verify.sh now exports the variable once, so every python it starts writes no bytecode in either checkout. The env stub logs it, and the harness clears it from the caller's env so verify.sh exec cannot satisfy the test. Removing the export fails test_fetch_downloads_a_missing_fixture_by_its_literal_name_then_locks_it in both layouts.
  • matplotlib: doctor in the sandbox with an empty HOME created ~/.config/matplotlib and ~/.cache/matplotlib/fontlist-v390.json, and nothing else. SKILL.md and the CLAUDE.md row list both.
  • GIT_OPTIONAL_LOCKS=0: test_doctor_leaves_a_stale_git_index_alone makes the index stat-dirty, runs doctor, and checks that the index bytes are unchanged. A plain git status afterwards must change them, which proves the setup is stale. Dropping the variable fails the test in both layouts (it passed the whole module in round 4).
  • Census: it now flags "${cmd[@]}" "$f", X=1 "${@}", if "$w"; then and while "$w"; do. A > inside [[ ]], (( )) or $(( )) is not a redirect, and a redirect after [[ ]] still counts. With round 4's regexes, the new harmless line [[ "$a" > "$b" ]] && echo $((a > b)); (( a > b )) is flagged, so every case fails. Without that line, exactly the 4 new forms fail.

Gates at bf6b7a1, all exit 0:

  • non-data pytest -n 8: 2238 passed, 4 skipped.
  • basedpyright: 0 errors, 0 warnings, 0 notes.
  • ruff_new_violations.py check and format against origin/dev: ok, 15 files.
  • F821/F822/F823/NPY201: passed.
  • red_green.py origin/dev: ok. test_doctor_leaves_a_stale_git_index_alone and the download test fail on base. The two fixture-copy tests are marked guards_existing_behaviour, because red-green's base run takes tests/ from the branch.

Round 4 (head 49a7c2f on dev 4de5cc0)

Safety as in round 3. Every run of verify.sh, its tests and its mutants used bwrap --ro-bind / / --bind <fake root> --tmpfs /tmp --unshare-net with the fake root under the worktree's .tmp. Gates used the overlay sandbox. The shared data/ was only read: it still has 2230 writable entries under codes, examples, io and issues.

Finding A (blocking): test_filtered_procar.py on locked fixtures. _copy_calc copied with shutil.copy, which keeps the 0444 mode, and the test then writes calc/PROCAR. It now uses writable_copy from tests.utils with an ignore that keeps the four VASP files. A repo-wide grep for shutil.copy, copy2, copytree, copyfile, cp, os.link and write_bytes(read_bytes()) finds no other fixture-copy site in the tests. write_bytes makes a new file with the default mode, tests/pyprocar/pyposcar/run.py is a script that pytest does not collect, and data_utils.copy_files has no caller.

The full data suite ran on a private reflink copy of data/{codes,examples,io,issues}. This PR's fetch locked the copy in a fake main checkout (fetch data/examples/*/* data/codes data/io data/issues). That left only the 9 container dirs writable. A linked worktree whose data links to that copy then ran pytest tests -m data -n 4 in an overlay sandbox:

-m data, -n 4, locked copy result
round-3 head a88f1f5 4 failed (test_selecting_one_orbital_by_name_raises_whichever_name_the_header_uses[x2-y2-dx2, dx2-x2-y2, dx2-dx2, x2-y2-x2-y2], PermissionError: [Errno 13] ... calc/PROCAR), 399 passed, 1 skipped
head 49a7c2f 403 passed, 1 skipped, 0 failed

Finding B: writes outside data/. These now appear in SKILL.md and the CLAUDE.md row, as listed under What changed. Each change was checked live:

  • A real download of data/examples/bands/atomic_levels ran in a network sandbox with HF_HOME in the fake root and an old planted log xet_20250101T000000000-0400_999999.log. Without the variable, hf_xet deleted the planted log. With HF_XET_LOG_DIR_DISABLE_CLEANUP=1, it kept it. Both runs created hub/.../refs, xet/logs/<new log>, and xet/<endpoint>/chunk-cache and staging.
  • In a scratch repo with a stat-only change, git status rewrote .git/index, and GIT_OPTIONAL_LOCKS=0 git status left it alone. git diff rewrote the index even with the variable, so SKILL.md names it for worktree-setup.
  • test_fetch_downloads_a_missing_fixture_by_its_literal_name_then_locks_it asserts that the stub env sees HF_XET_LOG_DIR_DISABLE_CLEANUP=1, and that in the main checkout it gets pixi run -q --locked -e. The test fails on the round-3 script (cleanup=unset) and passes at head.

Finding C: census gaps. See What changed. The new self-test covers 17 forms. Eleven mutants that insert one of the new forms into fetch are all killed: 2>, 1>, &>, >&file, exec 3>, sed -i, perl -pi, plain install, tar -x, unzip, and w=touch; $w.

Finding D: unlock. In the fake-root sandbox, a fixture held a hard link to an outside file with mode 0444, and fetch locked it. The round-3 unlock (! -type l) changed the outside file to -rw-r--r--. The new unlock (-type d) left it at -r--r--r--. In both cases rm -rf exited 0 and removed the fixture.

Finding E: new tests.

  • test_gc_keeps_the_work_of_a_run_in_progress_and_removes_a_finished_one plants two old runs. One has a live pid (pytest's own) and one a pid that has exited. The test kills the live()-always-false mutant. It passes on dev's script, so it is marked guards_existing_behaviour.
  • test_download_from_hf_stages_beside_the_fixture_not_in_the_system_tmp points tempfile.tempdir at a missing dir and asserts that the hub got a staging dir in <out>/data. It kills the TemporaryDirectory()-without-dir= mutant.

Mutants: 59 in all, 52 killed. The 7 survivors are the same 7 as in round 3: the require_resolved_below and real_path/physical_dir/is_below defence-in-depth checks. Only a concurrent swap inside data/ reaches them, and the root has scoped that out.

Gates at 49a7c2f, all exit 0:

  • non-data pytest -n 8: 2178 passed, 4 skipped.
  • basedpyright: 0 errors, 0 warnings, 0 notes.
  • ruff_new_violations.py check and format against origin/dev: ok, 10 files.
  • F821/F822/F823/NPY201: passed.
  • red_green.py origin/dev: ok.
  • full -m data -n 4 on the shared data/: 403 passed, 1 skipped.

Round 3 (head a88f1f5 on dev 4de5cc0)

Every execution of verify.sh, its tests and its mutants ran inside bwrap --ro-bind / / --bind <fake root> <fake root> --tmpfs /tmp --unshare-net. The fake root is one directory under the worktree's .tmp. Everything else, including /home, the worktree and the shared data/, was read-only. A probe inside that sandbox could not touch or chmod anything under /home/lllang. Each test's "outside" targets are directories inside its own fake root. No link points at / or at any path outside that root. The CI-equivalent gates ran in a second sandbox, with the filesystem read-only and the worktree an overlay whose writes go to an invisible tmpfs. That sandbox was needed because the library writes pyprocar.log into the working directory.

Finding A, replayed with the verifier's smallest layout (repro.sh): a fake main checkout with data/examples/bands/x/{PROCAR,sub/OUTCAR}, and symlinks at data/verify-runs/<stamp>-probe for the next 12 seconds, each pointing to an outside/<stamp>/ that holds .pid and work/calc/x/PROCAR. Then verify.sh run probe data/examples/bands/x driver.py:

[dev] verify.sh run exit: 0
[dev] outside tree CHANGED:        (.pid deleted, PROCAR overwritten, evidence/ and .start written outside)
[r2] verify.sh run exit: 2
[r2] outside tree CHANGED:         (.pid deleted, PROCAR overwritten, sub/OUTCAR, evidence/ and work/tmp created outside)
mkdir: cannot create directory '.../data/verify-runs/20261004-181147-probe': File exists
refusing: .../data/verify-runs/20261004-181147-probe already exists; rerun to get the next second's name
[head] verify.sh run exit: 2
[head] outside tree unchanged

New tests:

Test Round-2 head 70db097 origin/dev Head
test_run_refuses_a_run_dir_that_already_exists_and_changes_nothing, 5 layouts x 2 checkouts: a symlinked run dir, a real run dir, and a real run dir whose work, .pid or evidence links out. A date stub fixes the run name, and the whole fake tree (mode, type, link target, sha256) is compared before and after 10 fail: outside work/calc/x/PROCAR overwritten, or exit 0 10 fail pass
_refuses_everything now expects doctor to exit 2 (8 cases) fail, exit 0 fail pass
test_no_command_changes_anything_outside_data gives the fake worktree a pyprocar/ dir, so worktree-setup exits 0 and runs its cp, mkdir -p and ln -sfn lines. The test asserts the copied version, the data link target and .tmp pass fail pass
test_verify_sh_writes_only_on_the_lines_its_census_reviewed fail fail pass

The whole module: round-2 head 19 failed, 126 passed; origin/dev 105 failed, 40 passed; head 145 passed, 2 skipped. It also passes with LC_ALL=C, the locale CI runs in.

Mutants (47 single, written into the fake root and passed through VERIFY_SH_UNDER_TEST): 40 killed. The new ones:

  • Killed: mkdir -p "$run" (9 tests), the round-2 run block restored (11), the trap set before the run dir exists (5), subdirs made with mkdir -p (1), noclobber dropped from the .pid write (1, census lint only), doctor exiting 0 on a wrong root (8). Also killed: a cp, a > redirect, ln -sf, find -delete and tee added in fetch (1 to 5 each).
  • "chmod_below_data without its guard" still survives alone. On the fixed head no caller can reach chmod_below_data with an unresolved path, because run now refuses the planted layout at mkdir "$run", before the copy. Paired with the round-2 run block it dies (11 tests), and paired with mkdir -p "$run" it dies (9 tests). That pairing is the verifier's scenario.
  • The other survivors are guard layers backed by another layer: the two failed-cd mutants, two is_below mutants, scrub without its guard (all as in round 2), and the new require_resolved_below "$run" "$RUNS". A plain mkdir that succeeds always makes a real directory in the resolved verify-runs, so this check fires only if verify-runs is swapped during the run.

Gates at a88f1f5, each through verify.sh exec in the overlay sandbox:

  • non-data pytest: 2158 passed, 4 skipped.
  • basedpyright: 0 errors, 0 warnings.
  • ruff_new_violations.py check and format against origin/dev: ok (9 files).
  • F821/NPY201: ok.
  • red_green.py origin/dev: ok, 29 added or changed tests.
  • -m data for test_one_call_functions.py, test_unfold.py and test_download_examples.py: 57 passed. The shared data/ was read-only in the sandbox.

Rounds 1 and 2

All verify.sh runs below happen inside pytest's fake roots under the worktree's .tmp. Each fake root holds its own git repo, an outside/ tree (home/, tmp/runs/, an unenterable unenterable/) and a read-only victim/, and every symlink or hard link a test makes points into that tree. No test or mutant points at a real system path. The unenterable tests first assert that /pyprocar-288-absent, /codes, /examples and /verify-runs do not exist, because round-1 code would have mapped those names to /.

Findings, each with its test (both layouts, data in the repo and a linked worktree):

Finding Test Round-1 head 6587c87 Now
A test_an_unenterable_data_root_refuses_every_command (data dir mode 000; data linked to a mode-000 dir) fails: fetch exit 1 after a download attempt every command exits 2, nothing changes
B test_a_data_root_at_the_top_of_the_tree_refuses_every_command (data linked to the fake root's top, which stands in for /) fails: fetch data/outside locks outside/ refused, nothing changes
B test_a_worktree_data_link_spelled_with_a_leading_double_slash_still_locks passes passes; fails when the // normalisation is removed
outside root test_a_data_link_to_a_copy_outside_the_main_checkout_is_refused fails refused
C test_download_from_hf_refuses_a_prefix_without_downloading, ..._fetches_only_the_archive_named_exactly fail on origin/dev's library: data/c downloads codes.zip and codes-extra.zip; data/codes also pulls codes-extra.zip, deletes out/.cache and deletes the dataset folder in the shared cache pass
D test_fetch_leaves_a_hard_link_to_a_file_outside_data_writable fails: outside/home/f locked outside unchanged, the link named on stderr
E test_a_verify_runs_link_out_of_data_is_never_followed fails: clean deletes outside/tmp/runs/x/work clean, run, gc exit 2
F test_doctor_lists_the_fixture_roots_that_fetch_refuses_to_lock fails lists .hidden, escape, dangling, loop, runs, self, two words
G test_no_command_changes_anything_outside_data (every subcommand, every hostile argument) and test_verify_sh_changes_modes_only_in_chmod_below_data fail pass

Against round-1 head 6587c87d, 17 tests of tests/test_verify_fixtures.py fail and 118 pass; against origin/dev's verify.sh, 95 fail; against the final head, 135 pass and 2 skip (the // test applies to a worktree only) (.../against.py <rev> writes that revision's verify.sh to .tmp and points the tests at it through VERIFY_SH_UNDER_TEST, so no script is rewritten in place). red_green.py origin/dev: ok.

Mutants, each written to .tmp/mutants/<n>/verify.sh and passed through VERIFY_SH_UNDER_TEST:

  • Round 1's 19 mutants, ported to the new code: all killed.
  • Round-2 checks, all killed: // kept, data root not checked, is_below admitting the root itself, verify-runs link followed, clean through a symlinked work, chmod following hard links, doctor hiding unlockable roots, a raw chmod -R, an rm and an install -m added beside a guarded call in fetch.
  • Six mutants survive, because another layer refuses the same input: real_path ignoring a failed cd, physical_dir ignoring a failed cd, is_below by glob prefix, is_below admitting an empty parent, and chmod_below_data or scrub without require_resolved_below. A failed cd yields a path that cannot equal <main>/data and is never strictly below a non-empty canonical root, and a glob prefix equals the component test on canonical paths. Paired with the layer they back up, the guards show: clean any path alone fails 2 tests, with scrub's guard removed too it fails 8; no check of the real path fails 20, with chmod_below_data's guard removed too 24.

Real tool: download_from_hf at huggingface_hub 1.2.3 into a scratch root under the worktree's .tmp, with HF_HUB_CACHE pointed at an empty scratch dir. data/examples/bands/atomic (a prefix) raised data/examples/bands/atomic.zip is not in the lllangWV/pyprocar_test_data dataset after fetching 0 files. data/examples/bands/atomic_levels gave 15 files, no zip and no staging dir left. A pre-existing out/.cache/keep survived, and the scratch HF cache held only refs/main.

Gates at 70db097 on dev 5d6ae01, from the worktree with verify.sh exec: non-data pytest (2092 passed, 4 skipped), basedpyright (0 errors, 0 warnings), ruff_new_violations.py check and format (9 changed files, ok), the F821/NPY201 check, red_green.py origin/dev (ok; every added test fails on base except the three writable_copy tests, which carry the guard mark), and the -m data modules this PR edits (57 passed, read-only use of the shared data/).

Refs #288 (closes it once the operator has locked the shared data/)

@lllangWV
lllangWV force-pushed the read-only-fixtures branch 3 times, most recently from 5c125da to 70db097 Compare October 4, 2026 21:23
@lllangWV lllangWV changed the title feat(verify-pyprocar): make fetched fixtures read-only behind one path resolver feat(verify-pyprocar): make fetched fixtures read-only and keep verify.sh inside data/ Oct 4, 2026
@lllangWV

lllangWV commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

VERDICT: FAIL

Reviewed SHA: 70db097, on origin/dev 5d6ae01. The head is directly on top of dev, so no merge was needed.
Lane: Audit + regression (round 2). I read the full git diff origin/dev...70db097b and counted every line that chmods, deletes, moves, copies over or downloads in verify.sh, download_examples.py, verify_steps.py and tests/utils. Per the lane rules, I ran verify.sh only through the PR's own fake-root tests.

Commands and results

I worked in detached worktrees at the head and at origin/dev. I ran env binaries from the main checkout's .pixi/envs/dev with the worktree first on PYTHONPATH, the same as verify.sh exec. verify.sh itself was not invoked.

  • Non-data pytest (the CI command, -n 8): 2092 passed, 4 skipped, exit 0.
  • The PR's new modules (test_verify_fixtures.py, test_writable_copy.py, utils/test_download_examples.py): 140 passed, 2 skipped. The 2 skips are worktree-only cases in the data_in_repo layout.
  • basedpyright lock mode: 0 errors, 0 warnings. ruff_new_violations.py check and format: ok (9 files). ruff --select F821,F822,F823,NPY201: ok. gh pr checks 303: format, lint, red-green, test and typecheck all pass.
  • Full -m data suite, -n 4. A permission rule blocked the data symlink to the shared data/, so each worktree got a private reflink copy of data/{codes,examples,io,issues}. The shared store was only read.
    • Copy writable (the normal case): head 382 passed, 1 skipped; origin/dev 382 passed, 1 skipped. No regression.
    • Copy locked with a-w (the state after the operator's lock): head 382 passed, 1 skipped; origin/dev 2 failed and 2 errors (test_one_call_functions.py TestDosplot, TestFermiHandler and TestBandStructure2DHandler). This confirms acceptance item 4 for the test suite.
    • The changed driver examples/bands.py ran against the locked copy (a manual run that mirrors verify.sh run). Its legacy_bandsdosplot step passes. The 2 failing steps (obj_quiver_vectors_mode, obj_overlay_species) fail the same way on origin/dev, so they are unrelated to this PR. examples/parsers.py: all steps pass.
  • HF dataset listing (read only): every relpath in the notebooks has an exact <relpath>.zip. No caller in docs, notebooks or scripts depends on prefix matching. The only prefix callers, tests/scripts/download_test_data.py and the scripts/download.py epilog, already crashed on dev, as the PR body says.

Acceptance (#288)

  1. fetch locks fetched fixtures: done (chmod_below_data a-w, tests).
  2. run works on its writable copy: done (cp -RL then chmod_below_data u+w).
  3. doctor reports writable fixtures: done.
  4. every fixture-copy site works on read-only sources: done. Head passes the whole data suite on locked fixtures and origin/dev does not. The only remaining copy sites are the two writable_copy helpers.
  5. nothing outside data/ (or verify-runs) can be chmod-ed, copied over or deleted by any verify.sh command: claimed but false (finding 1). Every chmod and every rm -rf is correctly bounded, but run writes through a symlinked run directory without checking it.

Findings

  1. Blocking. verify.sh:195-198 (run) never checks that $run is a new, resolved directory. run="$RUNS/<timestamp>-<name>" is followed by mkdir -p "$run/evidence" "$run/work/tmp", echo $$ >"$run/.pid", the trap rm -f "$run/.pid" and cp -RL ... "$run/work/calc". All of these follow a symlink at $run. Scenario: an entry data/verify-runs/<YYYYMMDD-HHMMSS>-<name> is a symlink to a directory outside data/. This is the same planted-entry model the PR's tests use, for example verify-runs/run1/work -> outside/tmp. run then creates evidence/ and work/ in the target, overwrites same-named files under target/work/calc/, and overwrites target/.pid. chmod_below_data then refuses (exit 2), and the EXIT trap deletes target/.pid. I replayed those exact primitives in a scratch fake root (not verify.sh): the victim's work/calc/x/PROCAR was overwritten and victim/.pid was deleted. The lint test test_verify_sh_changes_modes_and_deletes_only_in_its_guarded_helpers allows that trap line, but the trap is not inside a guarded helper. No test covers a pre-existing run directory. Smallest fix: mkdir "$run" (no -p; it fails when $run exists, including as a symlink) before mkdir -p "$run/evidence" "$run/work/tmp". This also fixes the dev bug where two runs with the same name in the same second share one directory and the first run's trap removes the second run's .pid. Add a test that plants that symlink for the next few seconds.
  2. Non-blocking (doc truth). SKILL.md (new paragraph) and the CLAUDE.md row say a download "leaves the shared Hugging Face cache alone", and that verify.sh "writes only below the data root". This is not true: huggingface_hub 1.2.3 snapshot_download writes <HF cache>/datasets--lllangWV--pyprocar_test_data/refs/main even with local_dir (_snapshot_download.py:365-373). The PR body's own run saw this ("the scratch HF cache held only refs/main"). The write is harmless, but the statement should name it.
  3. Non-blocking (doc truth). SKILL.md says doctor, fetch, run, clean and gc "exit 2 unless data resolves to the main checkout's data/". doctor exits 0 (verify.sh:156, require_data || exit 0), and _refuses_everything asserts exit 0. SKILL.md says "plain names (letters, digits, _, . and -)", but PLAIN_NAME also forbids a leading . or -.
  4. Non-blocking (coverage). test_no_command_changes_anything_outside_data calls worktree-setup, but in the fake worktree that call stops at cp ... pyprocar/_version.py with exit 1, because the fake worktree has no pyprocar/ directory. As a result, mkdir -p "$MAIN/data" .tmp and ln -sfn "$MAIN/data" data never run under test. Those lines are unchanged and are the documented exceptions, but the test's allowed tuple suggests the author expected them to run. exec is untested by design.
  5. Non-blocking (lint scope). The lint pattern covers chmod, rm and similar commands, but not cp or > overwrites, ln -f, find -delete, truncate or tee. The three allowed lines are exactly the chmod and delete lines, not all the mutating lines. The sweep test is the real guard, and finding 1 shows a gap in it.
  6. Note (hardening). chmod_below_data uses find -P and scrub uses rm -rf, and neither stays on one filesystem. A bind mount inside data/ would take a lock or delete across it, because path checks cannot detect a mount. Consider find -P "$2" -xdev and rm -rf --one-file-system. Also, cp -RL in run follows a fixture's symlinks with no size bound (du -sm does not follow them). Real fixtures contain no symlinks and no hard links (checked read only on data/).
  7. Note (operator lock instructions in the PR body). These are safe: require_data refuses a symlinked or foreign data, and the shared data/ is a real directory with no hard links. The glob data/examples/*/* expands from the operator's working directory, though, so the instructions should say to run the command from the checkout root. All 38 current names are plain, so the loop's exit on the first refusal does not leave fixtures unlocked. No unlock or refresh path is documented. download_test_data(force=True) on a locked fixture will raise PermissionError.

The census found no other escape. require_data pins the data root to <physical main>/data and verify-runs to a real directory. real_path fails on a failed cd, on dangling links and on a link to a file. is_below compares component by component with no . or .., because every input is either PLAIN_NAME components or a fully resolved existing directory. require_resolved_below re-resolves before each chmod and delete, and uses exit rather than set -e in || contexts. scrub skips a symlinked work. Hard links are skipped. Empty, // and unset arguments are refused or abort under set -u. The download_examples.py rewrite stages next to the target, zipfile strips .. components and absolute paths, and the rewrite removes the two old rmtree calls that reached outside data/. The new tests stay inside tmp_path and use stubbed pixi and python and a fake hub, so they need no network.

@lllangWV

lllangWV commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

VERDICT: FAIL

Reviewed SHA: 70db097 (base origin/dev 5d6ae01; git merge-tree with origin/dev is clean). Round-1 head 6587c87 was used to reproduce the round-1 bugs.
Lane: live tests, red/green and CI gates (independent verifier, round 2).

Safety. Every verify.sh run (head, round 1, dev and mutants) ran inside bwrap. In E1 runs the whole filesystem is read-only except one sacrificial case directory under my worktree. In E2 runs that case directory is mounted as /, with /usr and /etc read-only. Each "outside" target is a directory inside that case directory, and every symlink stays inside it on the host. The few links to /, // and /// were created in a tmpfs that exists only inside the sandbox. I never pointed verify.sh at the shared data/. The data suite used a private reflink copy in a scratch clone.

Finding (blocking)

A pre-existing entry at the next run name sends verify.sh run outside data/. It overwrites and deletes files there. Code: .claude/skills/verify-pyprocar/scripts/verify.sh:195-200 and the trap at :198.

run builds run="$RUNS/$(date +%Y%m%d-%H%M%S)-$name", then runs mkdir -p "$run/evidence" "$run/work/tmp", echo $$ >"$run/.pid" and cp -RL "$fixture" "$run/work/calc". Nothing checks that $run did not exist before or that it resolves inside $RUNS. chmod_below_data refuses afterwards with exit 2, but the writes have already happened.

Smallest repro, in a fake main checkout with data/examples/bands/x/{PROCAR,sub/OUTCAR} and data/verify-runs/:

  1. For each of the next few seconds, create data/verify-runs/<YYYYmmdd-HHMMSS>-probe as a symlink to outside/<ts>/. That directory already holds .pid and work/calc/x/PROCAR.
  2. Run verify.sh run probe data/examples/bands/x driver.py.

Result: exit 2. outside/<ts>/.pid is overwritten and then deleted by the EXIT trap. outside/<ts>/work/calc/x/PROCAR is overwritten with the fixture's content. outside/<ts>/work/tmp, evidence/ and work/calc/x/sub/OUTCAR are created.

Variant: a real run dir at that name whose work, .pid or evidence is a symlink out of data/. That gives the same copy-over through work, and overwrites the target of .pid.

  • At dev 5d6ae01 the same layout changes 10 outside entries. The defect is pre-existing.
  • It falls inside the frozen acceptance item "nothing outside data/ (or inside verify-runs) can be chmod-ed, copied over or deleted by any verify.sh command".
  • It also contradicts the new CLAUDE.md row ("writes only below the data root ... never through a symlink").
  • No chmod escapes at head: chmod_below_data's guard stops it.
  • The trigger needs an entry planted at a predicted name. Ordinary use does not create one, so the severity is low, but the class is in scope.
  • Suggested fix: create the run dir with plain mkdir "$run", which fails when anything, including a symlink, exists at that name. Then mkdir "$run/evidence" "$run/work" "$run/work/tmp". This also stops two same-second runs with one name from sharing a directory.
  • Suggested test: add this layout to tests/test_verify_fixtures.py. That test would also kill the surviving mutant "chmod_below_data without its guard": on the planted layout that mutant doubles the damage, because it goes on to write evidence/ and new.txt outside.

Acceptance set (#288, frozen)

  • fetch locks fetched fixtures: done. I reflink-copied codes, io, issues and examples (24 GB apparent) into a scratch clone at the head and made the copy writable. fetch data/codes data/io data/issues data/examples/*/* exited 0. doctor then reported writable fixture paths: 0 and unlockable fixture roots: 0.
  • run works on its writable copy: done. I ran verify.sh run with the real library on the locked copy:
    • My probe driver wrote into $CALC. Writing or appending to the fixture raised PermissionError 13.
    • bands_plain.py exited 0.
    • In bands.py, the legacy_bandsdosplot step (writable_copy of a read-only dos fixture) passed. Its only two failures are the known crashes listed in features/bands.md:49-50.
    • clean then kept the evidence.
  • doctor reports writable fixtures: done. It counts the writable entries and lists hard-linked files. It lists 14 unlockable roots in my hostile layout: spaces, leading dash, unicode, .hidden, and dangling, looping and outward links.
  • every fixture-copy site works on read-only sources: done.
    • At head, pytest tests -m data -n 4 on the locked copy gave 382 passed, 1 skipped (80.9 s).
    • origin/dev's code on the same locked copy gave 378 passed, 2 failed, 2 errors. All four are PermissionError on copies of read-only fixtures in test_one_call_functions.py.
  • nothing outside data/ can be chmod-ed, copied over or deleted: not done. See the finding above. Every other layout passes; details below.

Live hostile layouts

live.py runs 36 layouts with about 110 commands each, then snapshot-compares the whole fake tree (mode, type, link target, sha256).

The commands are every subcommand with hostile arguments: the PR's REFUSED list plus data/tmp, data/home, data/w, -R, --, data/nl\nname, data/fïx, globs, .. paths, run names a b, -rf, ü and '', a driver path with a space, a newline and unicode, clean of links, outside dirs and odd run names, gc 0, gc -1 and gc 1e3, doctor, exec true and worktree-setup.

The layouts:

  • data linked to the fake top, to an unenterable dir, to a sibling copy, as // and /// spellings of the real data and of the copy, dangling, looping, and a data symlink in the main checkout
  • verify-runs linked out, dangling, looping or absent
  • symlinked work, work/calc and run dirs
  • hard links to outside files
  • a fixture holding outward links, plus an internal hard-link pair
  • a category dir linked out
  • a main checkout whose path contains -, a space, a newline and unicode, or ends in a newline
  • E2 cases: data unenterable, data linked to an unenterable dir, and data linked to /, // and ///

Results:

  • Head: 0 outside changes and 0 attempts beyond the case dir in 33 of 36 layouts. The three exceptions are the two planted-run layouts (the finding) and the pre-existing worktree-setup note below.
  • Round-1 bug A reproduced. In E2, r1 with an unenterable data dir chmod-ed /codes, /home, /outside and /examples at the sandbox root (54 entries). In E1 it chmod-ed its own case dir, which sits under the real /home; the read-only sandbox blocked everything else. Head refuses every mutating command with exit 2.
  • Round-1 bug B reproduced exactly. With data -> / (in tmpfs), r1's fetch data/tmp, data/home and data/codes locked the sandbox's /tmp, /home and /codes. Its find error shows //outside/.... Its gc 0 deleted /verify-runs/r/work. With // or /// it wrote /codesout, /runs and /self and deleted the same work dir. Head refuses all of these.
  • r1 also chmod-ed hard-linked outside files, and locked a sibling copy and a data-real behind a main-checkout data link. Head leaves hard links writable, names them on stderr and counts them in doctor.
  • // and /// spellings of the real data root are accepted at head and lock only inside data/.

Mutants

  • Owner's 35 mutants reproduced: 29 killed and the same 6 survive. The survivors are both "A" (failed cd), "is_below by glob prefix", "is_below admits an empty parent", "chmod_below_data without its guard" and "scrub without its guard".
  • Under my hostile layouts, five of the six change nothing beyond head itself. That confirms the layered-guard argument. The sixth is "chmod_below_data without its guard", which the planted-run layout kills.
  • My own mutants:
    • "require_resolved_below without equality": survives both the tests and the live layouts, because find -P never follows the start point.
    • "fixture_report without the shape filter": killed, 2 tests.
    • "chmod allows two links": killed, 5 tests, and live it chmods outside hard links.
    • "scrub rm -rf work/": killed, 5 tests, and live it deletes outside.

download_from_hf (fake snapshot_download only, HF_HUB_OFFLINE=1)

The fake also writes local_dir/.cache/huggingface metadata, as the real hub does.

  • Exact name: one call with allow_patterns=['<relpath>.zip'], local_dir is a staging dir inside data/examples/<cat>/, and only <relpath>/... is left.
  • Prefix (data/examples/dos/ne): FileNotFoundError ... ne.zip is not in the ... dataset, nothing extracted.
  • Network error mid-download: the staging dir is removed.
  • Existing fixture: returns early with no call.
  • Shared cache: never used (local_dir always set). out/.cache/keep survives every case.
  • Real network download: not checked, per the brief. The owner's real-hub run is unverified.

CI gates at head

Run through the main env, inside a sandbox where only the worktree is writable:

  • pytest tests -m "not data" -n 8: 2092 passed, 4 skipped.
  • basedpyright --project pyrightconfig.json: 0 errors, 0 warnings, 0 notes.
  • ruff_new_violations.py check origin/dev and format origin/dev: ok, 9 files.
  • ruff check --select F821,F822,F823,NPY201 .: passed. I added --extend-exclude .fake because my pytest basetemp lay inside the worktree.
  • red_green.py origin/dev: ok. 25 new tests fail on dev for the stated reasons and 3 are guarded with a reason.
  • gh pr checks 303: format, lint, red-green, test and typecheck all pass.

Claims

  • real_path/physical_dir fail on any failed cd: confirmed.
  • require_data accepts only <main>/data with a real verify-runs: confirmed. A dangling, looping or outward verify-runs is refused; an absent one is created.
  • // normalised: confirmed.
  • Component-wise is_below: confirmed.
  • download_from_hf fetches <relpath>.zip into staging, with no shared-cache or output/.cache rmtree: confirmed, fake hub only.
  • Hard links skipped and reported: confirmed.
  • RUNS resolved and scrub guarded: confirmed for clean and gc. run does not check the run dir itself (finding).
  • doctor lists unlockable roots: confirmed.
  • Behavioural test plus narrowed lint allowing exactly 3 lines: confirmed.

Notes (non-blocking)

  • worktree-setup writes through a symlinked worktree pyprocar/_version.py. cp overwrites the link's target outside data/ (verify.sh:245). This is pre-existing on dev and is worktree-setup's own file by design. cp --remove-destination would avoid it.
  • A main-checkout path with a newline makes run fail with exit 1. It fails at $(( $(du -sm "$fixture" | cut -f1) + MIN_FREE_MB )), where cut emits the path's tail, which then reaches arithmetic evaluation. This is pre-existing and fails loudly.
  • doctor truncates an unlockable name that contains a newline (data/nl\nname prints as data/nl). Cosmetic.
  • A library caller who passes glob characters in relpath downloads every matching archive into staging, then raises, and leaves an empty literal * directory. verify.sh's plain-name rule prevents this.
  • Local gotchas:
    • worktree-setup warns that the main checkout's pixi.lock differs from this branch's.
    • A pytest --basetemp inside the worktree makes the whole-repo ruff check . gate flag temp files.

@lllangWV
lllangWV force-pushed the read-only-fixtures branch from 70db097 to ebda932 Compare October 4, 2026 22:03
@lllangWV

lllangWV commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

VERDICT: FAIL

Reviewed SHA: a88f1f5. It sits directly on origin/dev 4de5cc0, and git merge-tree reports no conflict.
Lane: Audit + regression (round 3, independent verifier).

Safety. I did not run verify.sh outside the PR's own fake-root Harness. Every pytest run used bwrap --ro-bind / / with --unshare-net. The worktree was a tmp overlay, and the only writable directory was a scratch dir inside my worktree. The shared data/ was read-only in every run. My worktrees were set up by hand: _version.py copied, a data link to the shared store, and the env binaries taken from the main checkout's .pixi/envs/dev with the worktree first on PYTHONPATH. To test locked fixtures I made a private reflink copy of data/{codes,examples,io,issues} inside my worktree and ran chmod -R a-w on it inside bwrap. The shared data/ still has 2231 writable entries, the same as before.

Blocking finding

One fixture-copy site still fails on read-only fixtures (acceptance item 4 is claimed but not done). The site is tests/pyprocar/scripts/test_filtered_procar.py:39-44. There, _copy_calc copies each file with shutil.copy, which copies the mode bits. test_selecting_one_orbital_by_name_raises_whichever_name_the_header_uses (line 149-151) then calls procar.write_text(...) on that copy. Once the operator runs the PR's own lock command, the PROCAR copy is read-only, so the write raises PermissionError: [Errno 13] Permission denied: '.../calc/PROCAR'.

Repro: lock a reflink copy of the fixtures (chmod -R a-w), point the worktree's data at it, then run pytest tests -c .config/.pytest.toml --rootdir . -p no:cacheprovider -n 4 -m data.

-m data, -n 4 origin/dev head
shared data/ (writable modes, read-only mount) 403 passed, 1 skipped 403 passed, 1 skipped
locked private copy 6 failed, 2 errors 4 failed (all 4 test_selecting_one_orbital_by_name_... params), 399 passed, 1 skipped

The head fixes the 4 test_one_call_functions.py failures that origin/dev has on locked data. The test_filtered_procar.py site was not converted, and the PR's own data check covered only three modules. Fix: copy with shutil.copyfile, or apply the writable_copy chmod in _copy_calc.

Commands and results at head (all exit 0)

  • Non-data pytest -n 8: 2158 passed, 4 skipped.
  • basedpyright lock mode: 0 errors, 0 warnings.
  • ruff_new_violations.py check and format against origin/dev: ok, 9 files.
  • F821/F822/F823/NPY201: all passed.
  • red_green.py origin/dev: ok.
  • gh pr checks 303: 5/5 pass, on run 37239325339 at a88f1f5.
  • The PR's modules (test_verify_fixtures.py, test_writable_copy.py, utils/test_download_examples.py): 150 passed, 2 skipped, with no network. I repeated the run with /home and then /tmp as copy-on-write overlays whose upper dirs I could inspect. Outside the fake root, the only change was pyprocar.log in the worktree root, which conftest's import writes. Nothing was written to /tmp.

Census audit (verify.sh)

My own census of the redirections, cp, mv, ln, mkdir, rm, chmod, find -delete, tee, touch, here-docs, process substitution and traps in verify.sh found the same 18 lines as the PR table. Each one is new, strictly below the resolved data root, or the documented worktree exception. That holds on the error paths too: the trap is set only after mkdir "$run" succeeds, .pid is written with noclobber, require_resolved_below runs before every chmod and rm, find -P skips links and files with a second hard link, and rm -rf and rm -f do not follow links.

I added fake-root probes through the PR's harness fixture. All of them passed and changed nothing outside data:

  • clean with .pid and .start symlinked out, and with a hard-linked outside file in work.
  • gc with a hard-linked outside file in a read-only work.
  • run on a fixture that holds an outside hard link, or a link back into verify-runs.
  • fetch of a present or missing fixture under a category symlink that points out of data.
  • doctor on a wrong data root (exit 2).

Notes (not blocking)

  1. Writes the docs leave out. The CLAUDE.md row and SKILL.md say verify.sh writes only below the data root, with the listed exceptions (and refs/main). Writes made by the library and tools are missing from that list:

    • (a) doctor and the download path of fetch import pyprocar with cwd $REPO (verify.sh:17, :149, :174). The import opens pyprocar.log in append mode (log_utils.py:80-84, utils/__init__.py:19). I confirmed this in the sandbox.
    • (b) A real download goes through hf_xet. hf_xet writes ~/.cache/huggingface/xet/logs/xet_<ts>.log and uses ~/.cache/huggingface/xet/<endpoint>/chunk-cache. Its log-dir cleanup deletes old logs (HF_XET_LOG_DIR_MAX_RETENTION_AGE, ..._MAX_SIZE; HF_XET_LOG_DIR_DISABLE_CLEANUP turns it off). I saw the write attempt on the read-only home in the sandbox. The four existing logs in that dir show it happens on normal runs.
    • (c) git status in doctor may rewrite the index.
    • (d) In the main checkout, py is pixi run -e default without --locked, so it can install the env or rewrite pixi.lock. This is pre-existing, and it sits close to the operator's "never pixi install in the main checkout".

    None of these follows a planted path in data/. Still, the exception list is incomplete, and (b) is a delete outside data/ done by third-party housekeeping.

  2. The census lint has gaps. I checked each pattern against the PR's own WRITING_COMMAND, FILE_REDIRECT and QUOTED regexes. The lint does not see:

    • redirects with a file descriptor or &: 2>file, 1>file, &>file, >&file, exec 3>file
    • plain install src dst
    • sed -i
    • a command held in a variable (w="r""m"; $w f)
    • Python writes inside the inline py -c code
    • git writes
    • a new call to a reviewed function, or a reassignment of a variable that feeds a reviewed line

    It does see eval "rm …" and bash -c "chmod …". The behavioural sweep test is the real backstop, and chmod_below_data and scrub keep their own guards, so this matters less than it looks. The PR's claim that the lint catches any > to a file is still overstated.

  3. The unlock instruction is not the exact inverse of the lock (SKILL.md and the PR body). find -P … ! -type l -exec chmod u+w {} + also chmods files with a second hard link, which the lock skipped, and that link may be outside data/. rm -rf needs write permission only on directories, so find -P data/examples/<c>/<n> -type d -exec chmod u+w {} + is enough and cannot reach an outside inode.

  4. clean on a sibling name. clean data/verify-runs/$'r\n' scrubs the sibling run r, because $(…) in physical_dir strips trailing newlines. This stays inside verify-runs.

  5. Theoretical race. find -P … -exec chmod has a window between the check and the chmod: an entry swapped to a symlink in that window is followed. It needs a concurrent writer inside data/.

  6. The PR body's lock command works on the current shared store: every name is plain, and the store has no symlinks or hard links.

@lllangWV

lllangWV commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

VERDICT: FAIL

Reviewed SHA: a88f1f5, directly on origin/dev 4de5cc0 (git merge-tree clean, no merge needed). Round-1 head 6587c87 and round-2 head 70db097 were used to replay the earlier failures.
Lane: live tests, red/green and CI gates (independent verifier, round 3).

Safety. Every execution of verify.sh (head, r1, r2, dev, 57 mutants) and every test that drives it ran inside bwrap --ro-bind / / --bind <fake root> ... --tmpfs /tmp --unshare-net, where the only writable directory was a fake root under my worktree's .tmp. A probe inside it could not touch or chmod /home/lllang, the main checkout, its .git or the shared data/. The data -> /, // and /// replays ran in a second sandbox whose / is an empty tmpfs (only /usr and /etc bound read-only), so those links existed only inside it. The shared data/ was only read. The data suite used a private 24 GB reflink copy in a scratch clone.

Finding (blocking)

A fixture-copy site still fails on read-only fixtures. tests/pyprocar/scripts/test_filtered_procar.py:39-44 (_copy_calc) copies the fixture with shutil.copy, which keeps the 0444 mode, and line 151 then writes calc/PROCAR.

  • Repro: lock the fixtures with this PR's fetch (the command the PR body tells the operator to run after merge), then run pytest tests -m data. Four tests fail with PermissionError: [Errno 13] ... calc/PROCAR: test_selecting_one_orbital_by_name_raises_whichever_name_the_header_uses[x2-y2-dx2, dx2-x2-y2, dx2-dx2, x2-y2-x2-y2].
  • The same module passes 13/13 at head on the unlocked shared data/, so locking alone causes the failure.
  • The site arrived with fix(core): check projection axes against bands and structure, and give every parser its orbital names #298 (60fc0c0), which is in this PR's base but landed after round 2. Round 3's own data run covered only test_one_call_functions.py, test_unfold.py and test_download_examples.py.
  • This makes verify-pyprocar: make fetched fixtures read-only at the OS level #288 acceptance item "every fixture-copy site works on read-only sources" false at head.
  • Smallest fix: use writable_copy from tests.utils here, or shutil.copyfile, which does not copy the mode. A git grep for shutil.copy|copytree|copy2|copyfile finds no other fixture-copy site.

Acceptance set (#288, frozen)

  • fetch locks fetched fixtures: done. On the reflink copy, fetch data/examples/*/* data/codes data/io data/issues exited 0. doctor then printed writable fixture paths: 0 and unlockable fixture roots: 0, and exited 0. Only the 8 container dirs (data, data/examples, the categories) stay writable, by design. With the real hub (network sandbox, scratch HF_HOME), fetch data/examples/bands/atomic_levels downloaded the fixture and locked it (dr-xr-xr-x), and a rerun exited 0. fetch data/examples/bands/atomic raised FileNotFoundError: data/examples/bands/atomic.zip is not in the lllangWV/pyprocar_test_data dataset and left nothing behind.
  • run works on its writable copy: done. I ran verify.sh run with the real library on the locked copy.
    • A probe driver got PermissionError 13 on appending to $REPO/data/.../PROCAR and on creating a file in the fixture dir. It wrote into $CALC fine and parsed bands with shape (200, 20, 1).
    • bands_plain.py exited 0.
    • bands.py: 10 of 12 steps pass, including legacy_bandsdosplot (writable_copy of a read-only dos fixture). The 2 failures are the gaps documented in features/bands.md:49-50.
    • The other 9 drivers (parsers, dos, fermi2d, fermi3d, utilities, bs2d, ref_unfold, ref_orbits, ref_spin_ibz) hit no PermissionError. fermi2d's one failed step is the documented scalars_data gap.
    • clean and gc 0 removed work/ and kept evidence/.
  • doctor reports writable fixtures: done. It shows the count above. On a wrong data root it now exits 2, in all 10 such layouts I built.
  • every fixture-copy site works on read-only sources: not done. See the finding. Full -m data, -n 4, on the locked copy:
    • head: 399 passed, 4 failed (the finding), 1 skipped, in 64 s.
    • origin/dev: 395 passed, 6 failed, 2 errors, in 65 s. These are the same 4, plus the test_one_call_functions.py PermissionErrors that head fixes.
  • nothing outside data/ can be chmod-ed, copied over, overwritten or deleted: done for every static layout I built. Concurrent swaps can still break it; see notes 1 and 2.

Live replays and hostile layouts (head)

  • Round-2 failure (planted entry at the next run name, 70db097). 7 variants x 2 checkouts (data in repo, linked worktree): an absolute or relative symlink run dir, a dangling symlink, a regular file, and a real run dir whose work, .pid or evidence links out. I compared full snapshots (mode, type, link target, sha256).
    • r2 changed 9, 9, 8, 1 and 4 outside entries.
    • dev changed 10, 10, 5, 1 and 4.
    • Head exits 2 in all 14 and changes nothing.
  • Round-1 bug A (unenterable data dir, 6587c87). In the E1 sandbox, r1 locked the whole writable fake root (6230 entries, including other cases); the sandbox stopped it there. In the tmpfs-root sandbox it locked /codes, /home, /tmp and /outside (57 entries in repo layout, 69 in worktree layout). Head refuses fetch, run, clean, gc and doctor with exit 2 and changes nothing.
  • Round-1 bug B (data -> /, //, ///, tmpfs root only). r1 with / locked 83 entries (repo layout) or 70 (worktree layout) at the sandbox root. With // and ///, gc 0 deleted /verify-runs/r/work. Head refuses all 6 cases with exit 2 and changes nothing.
  • Battery. I ran 93 and 92 commands in the two layouts. Inputs were the PR's REFUSED list plus data/tmp, data/home, data/outside, data/self/codes, -R, --, a name with a newline, a unicode name and two words. Run names were a b, -rf, ü, '' and ../x. The tree also had outward and relative links, a hard link to an outside file, a symlinked run dir and work, gc 0, -1 and 1e3, doctor, exec and worktree-setup. Result: 0 outside changes. The hard-linked file stays writable and is named on stderr.

Mutants

  • The owner's 47 reproduce: 40 killed, and the same 7 survive.
  • Mine: 10 added, 7 killed. These survive:
    • V1, require_resolved_below without its equality check.
    • V5, live() always false. No test covers gc keeping an in-progress run.
    • V10, the chmod guard checks only is_below of the spelling.
  • None of the 10 survivors changes anything outside under the battery, planted, unenterable and tmpfs-root layouts (182 cases).
  • Under the race stubs below, require_resolved_below "$run" "$RUNS" removed (owner's survivor) does 12 outside changes against head's 1, and chmod_below_data without its guard (and V10) do 12 against 8. Only a race test, such as a date or cp stub that swaps verify-runs or the run dir, would kill them.
  • Library mutants: 6 of 7 killed. "Staging in the system tmp" (TemporaryDirectory() without dir=) survives. Nothing tests the staging location, which also matters because os.replace fails across filesystems.

CI gates at head (through verify.sh exec, sandboxed)

  • non-data pytest -n 8: 2158 passed, 4 skipped.
  • basedpyright: 0 errors, 0 warnings, 0 notes.
  • ruff_new_violations.py check and format origin/dev: ok, 9 files.
  • ruff check --select F821,F822,F823,NPY201 .: passed.
  • red_green.py origin/dev: ok. 29 tests: 26 fail on base for the stated reasons, and 3 are guarded with a reason.
  • gh pr checks 303: format, lint, red-green, test and typecheck all pass on a88f1f5.

Notes (non-blocking)

  1. Races I could construct. A stub on PATH ran each swap at a chosen point, standing in for a concurrent same-user process. Every one needs a process that swaps a directory in data/ for a symlink at the right moment, and no verify.sh command does that. Impact at head:
    • clean or gc: swapping the run dir between require_resolved_below and find makes scrub chmod u+w and then rm -rf an outside work/ tree, plus .start and .pid. gc shares scrub.
    • fetch: swapping the category dir between the guard and find locks an outside tree (chmod a-w, 4 entries).
    • run: swapping verify-runs before mkdir "$run" creates one empty dir outside, and then the run is refused.
    • run: swapping the run dir before cp -RL copies the fixture over an outside work/calc, and the EXIT trap deletes the outside .pid.
    • A real race without stubs: a watcher that swapped the run dir as soon as work/calc appeared (20k-file fixture) redirected cp mid-copy and made the trap delete the outside .pid.
  2. A driver that swaps its own run dir or $EVIDENCE makes verify.sh write outside. This needs no concurrency. After the driver, exit_code and side_effects.txt are overwritten through the swapped link, and the trap deletes the target's .pid. The census row admits that the driver can write anywhere itself. A cheap hardening is to call require_resolved_below "$run/evidence" "$RUNS" after the driver and in the trap.
  3. worktree-setup writes through a symlinked pyprocar/_version.py (it overwrites the target) or a symlinked pyprocar/ (it creates _version.py in the target). This is pre-existing and listed in the PR's Scope. cp --remove-destination fixes the first case. A symlinked .tmp points exec's TMPDIR outside, so tools write temp files there. A dangling .tmp makes worktree-setup and exec exit 1.
  4. Doc truth. CLAUDE.md and SKILL.md say a download writes only refs/main in the shared HF cache. With hf_xet installed (huggingface_hub 1.2.3), the real download also wrote $HF_HOME/xet/logs/*.log and $HF_HOME/xet/<endpoint>/chunk-cache/....
  5. The census lint misses some write forms: numbered-fd redirects (2>"$f"), &>"$f", exec 3>"$f", sed -i, perl -pi, install without -m, tar -x and unzip. The PR body says it catches "> or >> to a file". The sweep test remains the behavioural guard.
  6. fetch of a missing archive exits 1 with a Python traceback, while every other refusal exits 2 with a message.
  7. A second run with the same name in the same second is now refused with exit 2. The PR documents this.
  8. Inferred from the code, not run: gc 0 running beside a run can scrub that run's new work/ between mkdir "$run/work" and the .pid write, because GNU find -mmin +0 matches a dir created moments ago. The effect stays inside verify-runs, and dev has the same ordering.
  9. Local gotcha: worktree-setup warns that the main checkout's pixi.lock differs from this branch's.

@lllangWV

lllangWV commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

VERDICT: FAIL

Reviewed SHA: 49a7c2f (round 4). The base in the spawn brief was origin/dev 4de5cc0. During this review origin/dev moved to f93c8bd (#304). git merge-tree reports no conflict with either.
Lanes: both (live + red/green + gates, and audit + regression), independent verifier.

Safety. Every run of verify.sh, its tests, its mutants and the data suite used bwrap --ro-bind / / --bind <fake root> --dev /dev --proc /proc --tmpfs /tmp --unshare-net. The fake root was one directory inside my worktree. A probe inside that sandbox could not write, append to or chmod /home/lllang, ~/.bashrc, the main checkout, its .git, the shared data/ or a fixture file. Gates ran with a tmpfs overlay on the worktree on top of that. All fixture work used a scratch clone inside the fake root, holding a private reflink copy of data/{codes,examples,io,issues} (24 GB). The shared data/ was only read, and it still has 2230 writable entries.

Blocking finding

Once merged into current dev, a fixture-copy site fails on locked fixtures. Acceptance item 4 is false for the result of the merge.

  • The site is tests/pyprocar/io/elk/test_parser.py:862-867, in test_real_elk_6_3_bands_beside_a_task_10_elmirep_keep_their_l_m_names. It arrived with fix: resolve the #285 follow-ups (DOS labels, autobandsplot report, BS2D grid, overlay weights, Elk lmirep) #304 (f93c8bd) on dev after this PR's base. shutil.copytree(ELK_BANDS_SP, calc_dir, ...) keeps the 0555 dir mode. shutil.copy(.../dos/ELMIREP.OUT, calc_dir) then raises PermissionError: [Errno 13] ... bands/ELMIREP.OUT.
  • Repro: merge origin/dev into 49a7c2f. Lock a reflink copy of the fixtures with this PR's fetch. Run pytest tests -m data -n 4. Result: 1 failed, 405 passed, 1 skipped.
  • The same test passes (1 passed) on an unlocked copy in the same merged tree, so the lock alone causes the failure.
  • The PR's Scope tells the operator to lock data/ after merge, so as written the merge plus the lock turns -m data red.
  • Smallest fix: rebase on dev, copy with writable_copy(ELK_BANDS_SP, calc_dir, ignore=...) from tests.utils, then rerun -m data on a locked copy.
  • My grep of the merged tree for copytree, shutil.copy*, cp -R/-a, os.link and write_bytes(read_bytes()) finds one more new site, tests/pyprocar/scripts/test_auto_bandsplot.py:52. It passes when locked, because it writes nothing into the copy.

Acceptance set (#288), at 49a7c2f

  • fetch locks fetched fixtures: done. fetch data/examples/*/* data/codes data/io data/issues exited 0 in the scratch clone. Only the 9 container dirs stay writable.
  • run works on its writable copy: done (unchanged this round). The planted-run replay is below.
  • doctor reports writable fixtures: done. On the shared store it prints writable fixture paths: 2223 and unlockable fixture roots: 0, and exits 0.
  • every fixture-copy site works on read-only sources: done at head, not done after merge (the finding above). Full -m data, -n 4, on the locked private copy:
tree result
a88f1f5 4 failed (test_selecting_one_orbital_by_name_...[x2-y2-dx2, dx2-x2-y2, dx2-dx2, x2-y2-x2-y2], PermissionError ... calc/PROCAR), 399 passed, 1 skipped
49a7c2f 403 passed, 1 skipped, 0 failed
49a7c2f merged with f93c8bd 1 failed (Elk, above), 405 passed, 1 skipped

My own grep of the head finds no unconverted site. verify_steps.writable_copy and tests.utils.writable_copy are the copy helpers, verify.sh:202 is followed by chmod_below_data u+w, and data_utils.copy_files has no caller.

  • nothing outside data/ is chmod-ed, copied over, overwritten or deleted: done for every static layout I built. Two of the documented writes were not re-run live by me, and the list misses a cache (note 1).

Round-4 changes, checked

  • _copy_calc uses writable_copy with an ignore that keeps the wanted names. The module passes on the locked copy, and its 4 tests fail at a88f1f5 as expected.
  • The HF_XET_LOG_DIR_DISABLE_CLEANUP=1 prefix on the py function reaches the child. The stub sees cleanup=1. The name exists in the hf_xet 1.2.1 binary, next to ..._MAX_RETENTION_AGE and ..._MAX_SIZE. Removing the prefix is killed (2 tests).
  • py with --locked: removing it is killed (1 test, main-checkout layout).
  • GIT_OPTIONAL_LOCKS=0 git status was tested live in a scratch repo with a stat-dirty index. With the variable, .git/index is untouched. Without it, the index is rewritten.
  • The unlock instruction was tested in the sandbox. A fixture held hard links to an outside 0444 file and an outside 0644 file, plus a symlink to the 0444 file, and head's fetch locked it. The new -type d unlock plus rm -rf exited 0 and removed the fixture. Both outside files kept their mode and sha256; only their link count fell 2 to 1. The round-3 ! -type l unlock changed the 0444 file to 0644.
  • The new gc test kills live() always false, live() always true, live() reduced to the .pid file check, and the gc live check removed (2 failures each, on the right asserts).
  • The staging test kills TemporaryDirectory() without dir= and TemporaryDirectory(dir=output_path).
  • The planted-run-dir replay ran with 7 kinds x 2 layouts (main, linked worktree), next 13 seconds planted. The kinds were an absolute or relative symlink, a dangling symlink, a file, and a real dir whose work, .pid or evidence links out. origin/dev 4de5cc0 exits 0 and changes the outside tree in 10 of 14 (18, 6, 2 or 8 diff lines). Head exits 2 with "already exists" in all 14, and the outside tree is unchanged.
  • Census self-test: the new forms (N>, &>, >&file, >|, exec {fd}>>, sed -i/--in-place, perl -pi, install, tar, unzip, $w, "$w", "${w}", && "$@") are flagged, and its harmless line is not. Gaps are in note 2.

Gates

At 49a7c2f, through verify.sh exec in the overlay sandbox, all exit 0:

  • non-data pytest -n 8: 2178 passed, 4 skipped.
  • basedpyright: 0 errors, 0 warnings, 0 notes.
  • ruff_new_violations.py check and format: ok.
  • F821/F822/F823/NPY201: passed. The first run flagged 13 files, all pytest temp files in my own scratch dir. With that dir excluded it passed.
  • red_green.py origin/dev: ok.

On the head merged with f93c8bd (scratch clone, same env):

  • non-data pytest: 2226 passed, 4 skipped.
  • basedpyright: 0/0/0.
  • ruff check and format against origin/dev: ok, 10 files.
  • F821 set: passed.
  • red_green: ok.

gh pr checks 303: format, lint, red-green, test and typecheck all pass on 49a7c2f (run 37242827066).

Notes (non-blocking)

  1. The list of writes outside data/ is still incomplete. I ran doctor in the sandbox with an empty HOME. Its import matplotlib (and pyprocar's own import) created $HOME/.config/matplotlib and wrote $HOME/.cache/matplotlib/fontlist-v390.json. matplotlib rewrites that file when it is stale, and a download in fetch makes the same import. Inferred from verify.sh:24: in the main checkout py runs without PYTHONDONTWRITEBYTECODE, so python can write __pycache__ in the main checkout's pyprocar/. Both are cache writes, but the round-4 claim is that SKILL.md and CLAUDE.md list every write.
    • Confirmed: the pyprocar.log write (observed live), the refs/main write (_snapshot_download.py:369-376 at 1.2.3; .locks is used only on the cache path, not with local_dir), and the hf_xet variable name (binary).
    • Not re-run live by me: the hf_xet log, chunk-cache and staging writes. That needs a real download through fetch, which I could not route through the shared data/ safely.
  2. The census lint misses some write forms.
    • Inside its own "command held in a variable" class it misses "${cmd[@]}" "$f", "${@}", if "$w"; then, and nice or timeout "$w".
    • Outside that class it misses a redirect inside a bash -c or eval string, awk -i inplace, gzip/gunzip, sort -o, curl -o, wget -O, link, patch, split, zip, ex, find -fprint and git writes.
    • It flags these harmless lines: [[ "$a" > "$b" ]], echo $((a > b)), (( a > b )).
    • The behavioural sweep killed all 4 census-evading mutants I added to fetch (4 to 6 tests each). Those were an array-held c"p" into the repo and into outside/, an eval redirect into the repo, and a py -c open(...,'w').
  3. Dropping GIT_OPTIONAL_LOCKS=0 survives the whole module (164 passed). No test pins it, though the behaviour is true live.
  4. The gc test treats an exited child's pid as finished, so a pid reuse would flake it. The odds are negligible.
  5. _copy_calc now skips a wanted file that is missing from the source without an error, where shutil.copy raised. Today every caller's file exists.
  6. Inferred from code: if data/examples/<category> were a symlink out of data/, the operator's find -P ... -type d unlock and rm -rf would follow it, because only the last component is not followed. fetch refuses that layout, and doctor lists the roots under it as unlockable, so the operator sees the problem first.

copytree keeps a read-only fixture's modes, so a test that writes into
its copy fails with PermissionError once fixtures are locked.
writable_copy dereferences symlinks and adds user write to the copy
only.
… its own copies

Run a copy of the harness in scratch repos (data in the repo, data
linked, and a linked worktree) with a stub env. Cover the round-2
escapes: a symlinked fixture given to run, a symlinked work/ given to
clean and gc, glob characters given to fetch, and a fixture root that
is a symlink.
…th resolver

fixture_path turns an argument into the real path of a fixture inside
data/ or refuses it: plain names only, so no glob reaches the Hugging
Face pattern, and the resolved path must be a fixture path. fetch, run
and doctor act only on that path. run copies with cp -RL, so its copy
holds no symlink. Every chmod runs through find -P ! -type l, which
never follows a symlink. fetch needs the env only to download.
…hostile data roots

Unenterable data, data at the top of the tree, data linked to a copy,
a leading // spelling, a verify-runs link out of data/, hard links to
outside files, and unlockable roots in doctor. A sweep runs every
command against a sacrificial tree and compares it before and after.
download_from_hf fetches only the archive named exactly.

Refs #288
real_path fails on any failed cd and drops a leading //. Every command
that touches data/ first requires data to resolve to <main>/data and
verify-runs to be a directory in it, compares paths by component, and
chmods only through chmod_below_data, which skips symlinks and files
with a second hard link. doctor lists roots fetch cannot lock.
download_from_hf fetches exactly <relpath>.zip and no longer deletes
<output>/.cache; fetch points its HF cache into data/verify-runs.

Refs #288
…ed HF cache

download_from_hf deleted the dataset's folder from the active Hugging
Face cache after every download. snapshot_download with local_dir set
to a staging dir beside the target touches no shared cache, and the
archive path is spelled once.

Refs #288
…ctor list printer

scrub requires a resolved run dir below verify-runs before rm -rf, the
lint now covers rm and mv too, fetch no longer redirects HF_HUB_CACHE,
and the tests share one env-and-driver setup.

Refs #288
…tor exits 2 on a wrong data root

Plants a symlink, a real dir, and real dirs with symlinked work, .pid or
evidence at the next run names, and snapshots the whole fake tree.
The sweep test now gives the fake worktree a pyprocar/ dir, so
worktree-setup reaches its mkdir and ln lines.
run named its dir <timestamp>-<name> and wrote into it with mkdir -p,
a .pid redirect and cp -RL without checking it was new, so an entry
planted at that name (a symlink, or a run dir whose work, .pid or
evidence links out of data/) made run overwrite and then delete files
outside data/ before chmod_below_data refused. A plain mkdir now
creates the dir and fails on any existing entry, the dir must resolve
strictly below verify-runs, its subdirs use plain mkdir, and noclobber
guards every redirect. This also stops two same-second runs with one
name from sharing a dir.

doctor exits 2, like the other data commands, when the data root is
wrong. The lint now lists every line of verify.sh that writes, links,
moves or deletes, not only chmod and rm.
…he run-dir rule and how to refresh a fixture
…t one stamped run in the test

A date stub fixes the run name, so the planted-run test plants one
entry instead of guessing the next 31 seconds, and the post-driver
writes into evidence keep their old behaviour.
…us lines under 100 columns

CI runs in an ASCII locale, where mkdir's quoted error message failed
to decode. Two long verify.sh lines are split so the census lint pins
them without implicit string concatenation.
…hrough pixi run --locked in the main checkout
…i --locked, and stop doctor's git status rewriting the index
… edits, install, tar, unzip and a command held in a variable
…sage, and mark the census self-test as guarding
…able_copy

The check reads tests/ and flags shutil.copy, copy2, copyfile, copytree,
copymode, copystat, os.link, Path.hardlink_to, and cp or rsync through
subprocess or os.system, when an argument holds a path built from DATA_DIR
or the repo root joined with data. It follows names, attributes, defaults,
call-site arguments, return values, pytest fixtures and parametrize values,
across modules. It flags #304's Elk test and the auto bandsplot fixture.

The module-name helpers move from test_user_warnings into tests/utils so
both checks share them.
…ixtures stay usable

#304's Elk test copied a 0555 bands dir with copytree, then copied ELMIREP.OUT
into it, which raised PermissionError on locked fixtures. writable_copy now
also copies a single file.
…BYTECODE=1

py ran pixi run without it, so doctor, a download and a run driver rewrote
stale __pycache__ files in the main checkout (measured in an overlay sandbox).
The env stub now logs the variable, and the harness clears it so the
caller's env cannot satisfy the test.
Dropping GIT_OPTIONAL_LOCKS=0 from doctor's git status survived the suite.
The test makes the index stale, runs doctor, and checks the index bytes; a
plain git status afterwards must rewrite it, so the setup is proven stale.
…"$w", and ignores [[ ]] and arithmetic comparisons

A > inside [[ ]], (( )) or $(( )) is a comparison, not a redirect.
…s, and add the fixture-copy rule

The rule table names tests/test_fixture_copies.py as the enforcer of
copying fixtures only through tests.utils.writable_copy.
… and map call arguments to positional parameters only
… untyped for from_code, and wrap two long self-test lines
@lllangWV
lllangWV force-pushed the read-only-fixtures branch from 49a7c2f to 5222718 Compare October 4, 2026 23:54
…hardlink first, fixture registration in the bind loop) and document writable_copy's file form
@lllangWV

lllangWV commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

VERDICT: PASS+NOTES

Reviewed SHA: bf6b7a1 (round 5). origin/dev is still 499b1bc, and it is an ancestor of the head, so the head is the merged result. Nothing needed merging.
Lanes: both (live + red/green + gates, and audit + regression). Independent verifier.

Safety. Every command that ran verify.sh, its tests, its mutants or the data suite ran inside bwrap --ro-bind / / --dev /dev --proc /proc --tmpfs /tmp --unshare-net. The writable part was either one fake root under my worktree's .tmp, or a tmpfs overlay on my worktree that discards its writes. I ran a probe inside the sandbox first. It could not touch, append to or chmod /home/lllang, ~/.bashrc, the main checkout, its .git, the shared data/ (a dir and a fixture file), data/verify-runs or the owner's worktree. The fake root was writable. The shared data/ was only read, and it still has 2230 writable entries.

Acceptance set (#288)

  • fetch locks fetched fixtures: done. I made a fake main checkout holding the head's verify.sh and a private reflink copy of data/{codes,examples,io,issues}. In that copy, fetch data/examples/*/* data/codes data/io data/issues exited 0 and printed 41 read-only: lines. Before the fetch, 2232 entries were writable. After it, 9 were, and those are the container dirs only (data, data/examples, the categories and verify-runs).
  • run works on its writable copy: done. The round-2 planted-run-dir replay used 7 kinds in 2 layouts (data in the repo, and a linked worktree). The kinds are an absolute or relative symlink, a dangling symlink, a regular file, and a real run dir whose work, .pid or evidence links out.
    • Head: all 14 exit 2 with "already exists", and nothing outside changes.
    • origin/dev: 10 of 14 exit 0 and change 10, 10, 5, 1 or 4 outside entries.
    • Round-2 head 70db097: 10 of 14 change 9, 9, 8, 1 or 4 outside entries.
  • doctor reports writable fixtures: done. On the copy, doctor printed writable fixture paths: 2223 before the lock. After it, it printed writable fixture paths: 0 and unlockable fixture roots: 0, and exited 0.
  • every fixture-copy site works on read-only sources: done, on the merged result.
  • nothing outside data/ is chmod-ed, copied over, overwritten or deleted: done for the layouts I ran.
    • Round 5 changes verify.sh only by export PYTHONDONTWRITEBYTECODE=1, which replaces the two per-command prefixes.
    • The sweep test passes, and the planted replay is clean (above).
    • Round 4's hostile-layout battery is unaffected by this delta, and I did not re-run it.

Round-5 claims

  • The new check fails on both historical sites and passes on the head: confirmed. I ran head's raw_fixture_copies on scratch copies of head's tests/:
    • head: 0 hits.
    • With f93c8bdf:tests/pyprocar/io/elk/test_parser.py swapped in: 2 hits (lines 862 copytree and 867 copy).
    • With 60fc0c02:tests/pyprocar/scripts/test_filtered_procar.py swapped in: 1 hit (line 43, reached through the src=CALC default).
    • With origin/dev's test_auto_bandsplot.py swapped in: 1 hit (line 52).
  • writable_copy file form: confirmed. shutil.copyfile gives the new file the default mode. test_writable_copy.py passes 4/4, and the Elk site passes on the locked copy.
  • Moved helpers leave test_user_warnings.py behaviour unchanged: confirmed.
    • module_parts, module_name and imported_from in tests/utils/ast_modules.py match the removed private copies token for token.
    • I loaded both modules side by side. On all 284 .py files under pyprocar/ and tests/, with 3 roots, the helpers return identical results.
    • loud_user_logger_calls, user_logger_lowered_in and the exported-name sets return identical hits. The only difference is how paths print, because my harness loaded dev's file from a scratch location.
    • The head module passes 5/5, and dev's module passes 5/5 in the head tree.
  • PYTHONDONTWRITEBYTECODE=1 export: confirmed. Removing it fails test_fetch_downloads_a_missing_fixture_by_its_literal_name_then_locks_it in both layouts.
  • GIT_OPTIONAL_LOCKS=0 is pinned: confirmed. Dropping it fails test_doctor_leaves_a_stale_git_index_alone in both layouts.
  • Census: confirmed. I added "${cmd[@]}" "$real", if "$w"; then :; fi and X=1 "${@}" to fetch, one at a time. Each fails test_verify_sh_writes_only_on_the_lines_its_census_reviewed. The line [[ "$rel" > "a" ]] && (( 1 > 0 )) && : $((2 > 1)) passes all 171 tests.
  • Docs: confirmed. SKILL.md and the CLAUDE.md row now list matplotlib's ~/.config/matplotlib and its font cache, and say that PYTHONDONTWRITEBYTECODE applies in both checkouts.
  • Not re-checked: "fourteen mutants of the check are all killed" and "22 flagged forms".

Gates at bf6b7a1 (= merged), in the overlay sandbox through verify.sh exec, all exit 0

  • non-data pytest -n 8: 2238 passed, 4 skipped.
  • basedpyright: 0 errors, 0 warnings, 0 notes.
  • ruff_new_violations.py check and format against origin/dev: ok, 15 files each.
  • ruff check --select F821,F822,F823,NPY201 .: all checks passed.
  • red_green.py origin/dev: ok.
  • test_verify_fixtures.py: 171 passed, 2 skipped.
  • gh pr checks 303: format, lint, red-green, test and typecheck all pass on bf6b7a1 (run 37245957425).

Notes (non-blocking)

  1. The static check has gaps for plausible copy forms. Each case below is a single synthetic tests/test_*.py added to a scratch copy of head's tests/, one at a time. "Missed" means 0 hits.
    • Missed: the repo's own parametrized-fixture pattern. @pytest.fixture(params=ALL_TEST_CASES, ids=...) returns request.param, and the test then calls shutil.copytree(calc_info.path, tmp_path / "c"). This is the shape of tests/pyprocar/core/test_fermisurface.py:187. A fixture whose params=[DATA_DIR / ...] returns request.param is missed in the same way. The parametrize("case", ALL_TEST_CASES) form is flagged.
    • Missed: a copy helper in another module. tests/helpers_mod.py: def copy_any(src, dst): shutil.copytree(src, dst), called as copy_any(DATA_DIR / ..., tmp). Call arguments are followed only into same-module functions.
    • Missed: request.getfixturevalue("calc_dir"), where calc_dir is a fixture that returns a DATA_DIR path.
    • Missed: an alias of the repo root. All of these are missed: REPO = Path(__file__).resolve().parents[1]; REPO / "data" / ..., TEST_DIR.parent / "data", Path(__file__).parents[1].joinpath("data", ...) and os.path.join(os.path.dirname(...__file__...), "data"). The seed recognises only <expr containing __file__ or ROOT_DIR> / "data".
    • Missed: other plausible forms.
      • A class-body attribute read through self: class T: calc = DATA_DIR / ... with shutil.copytree(self.calc, ...). Setting self.calc = CALC in a method is flagged.
      • cmd = ["cp", "-a", str(CALC), dst]; subprocess.run(cmd).
      • A method that returns a fixture path.
      • from shutil import * with copytree(...), and cp = shutil.copytree; cp(CALC, ...).
      • @pytest.fixture(name="calc").
      • functools.partial(shutil.copytree).
      • shutil.make_archive plus tarfile.extractall, which keeps the modes.
    • Flagged: a fixture built from request.param, a conftest fixture, a dict subscript of tainted values, and parametrize with an imported list.
    • Fine by design: write_bytes(read_bytes()).
    • The PR body and the CLAUDE.md row say the check "follows ... pytest fixtures, parametrize values and imports". That is true for the forms its self-test covers, but not for fixture params= or for calls into helpers in other modules.
    • None of these forms exists in tests/ today. The behavioural backstop is the data suite on locked fixtures, which passes. Cheapest additions:
      • Treat a fixture whose decorator's params= holds a fixture path as a fixture path.
      • Taint class-body names read as self.<name> or cls.<name>.
      • Seed any name assigned from an expression that contains __file__.
  2. Local gotcha: worktree-setup warns that the main checkout's pixi.lock differs from this branch's.

@lllangWV
lllangWV merged commit ce6a550 into dev Oct 5, 2026
5 checks passed
@lllangWV
lllangWV deleted the read-only-fixtures branch October 5, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant